Reinforcement Learning-based Control Research Articles

Adaptive reinforcement learning-based control using proximal policy optimization and slime mould algorithm with experimental tower crane system validation

This paper presents a novel optimal reference tracking control approach resulted from the combination of a popular policy gradient Reinforcement Learning (RL) algorithm, namely Proximal Policy Optimization (PPO), and a metaheuristic Slime Mould Algorithm (SMA). One of the most important parameters in the PPO-based RL process is the learning rate, which has a big impact on how the parameters of the actor neural network (NN) are iteratively updated. In every episode of the RL process, the weights and the biases of the actor NN are multiplied with the learning rate, determining how much the learning agent will step into a certain direction computed based on previous experiences. The classical PPO algorithm usually relies on fixed values for the learning rates which rarely change, or not at all, during the learning process. However, its main drawback is that the learning agent cannot take advantage of positive momentum in the learning process by accelerating towards good learning experiences or slow down and quickly change the direction in the case of consecutive negative learning experiences. The main objective of the combination proposed in this paper is to create an adaptive SMA-based PPO approach applied to control systems, which instead of using fixed learning rate values, it uses the SMA to compute optimal values of the learning rates in each time step of the learning process based on the progress of the learning agent. This paper investigates if the adaptive SMA-based PPO control approach can be considered as an alternative to the classical PPO version, which employs fixed values of the learning rate. A comparison is carried out using control system performance indices gathered while performing an optimal reference tracking control task on tower crane system laboratory equipment.

A novel investigation on the effects of state and reward structure in designing deep reinforcement learning-based controller for nonlinear dynamical systems

A novel investigation on the effects of state and reward structure in designing deep reinforcement learning-based controller for nonlinear dynamical systems

Guest Editorial Special Issue on Reinforcement Learning-Based Control: Data-Efficient and Resilient Methods

Guest Editorial Special Issue on Reinforcement Learning-Based Control: Data-Efficient and Resilient Methods

Reinforcement learning-based control for waste biorefining processes under uncertainty

Waste biorefining processes face significant challenges related to the variability of feedstocks. The supply and composition of multiple feedstocks in these processes can be uncertain, making it difficult to achieve economically feasible and sustainable waste valorization for large-scale production. Here, we introduce a reinforcement learning-based framework that aims to control these uncertainties and improve the efficiency of the process. The framework is tested on an anaerobic digestion process and is found to perform better than traditional control strategies. In the short term, it achieves faster target tracking with increased precision and accuracy, while in the long term, it shows adaptive and robust behavior even under additional seasonal supply variability, meeting downstream demand with high probability. This reinforcement learning-based framework offers a promising and scalable solution to address uncertainty issues in real-world biorefining processes. If implemented, this framework could contribute to sustainable waste management practices globally, making waste biorefining processes more economically viable and environmentally friendly.

Home electricity data generator (HEDGE): An open-access tool for the generation of electric vehicle, residential demand, and PV generation profiles

In this paper, we present the Home Electricity Data Generator (HEDGE), an open-access tool for the random generation of realistic residential energy data. HEDGE generates realistic daily profiles of residential PV generation, household electric loads, and electric vehicle consumption and at-home availability, based on real-life UK datasets. The lack of usable data is a major hurdle for research on residential distributed energy resources characterisation and coordination, especially when using data-driven methods such as machine learning-based forecasting and reinforcement learning-based control. We fill this gap with the open-access HEDGE tool which generates data sequences of energy data for several days in a way that is consistent for single homes, both in terms of profile magnitude and behavioural clusters.•From raw datasets, pre-processing steps are conducted, including filling in incomplete data sequences, and clustering profiles into behaviour clusters. Transitions between successive behaviour clusters and profiles magnitudes are characterised.•Generative adversarial networks (GANs) are then trained to generate realistic synthetic data representative of each behaviour groups consistent with real-life behavioural and physical patterns.•Using the characterisation of behaviour cluster and profile magnitude transitions, and the GAN-based profiles generator, a Markov chain mechanism can generate realistic energy data for successive days.

Deep Reinforcement Learning-Based Control for Asynchronous Motor-Actuated Triple Pendulum Crane Systems With Distributed Mass Payloads

For transporting large cargoes in practice, the triple pendulum hoisting pattern is widely utilized, where the distributed mass payload (DMP) is suspended under the hanger, the hanger is suspended on the hook, and the hook is connected with the hoisting mechanism. However, existing studies mainly focus on single pendulum and double pendulum hoisting patterns, and few of them pay attention to triple pendulum hoisting patterns. Compared with single pendulum and double pendulum cranes, triple pendulum cranes, which control 4 degrees of freedom (DoFs) with merely 1 control input, have a higher underactuation degree, with higher control difficulties. In addition, the nonlinear characteristics of alternating current (AC) asynchronous motors, such as unknown nonlinear dead zones and time delays, will degrade tracking accuracy and control performance. To solve the above problems, a model-free deep reinforcement learning-based trajectory planning method is proposed for triple pendulum DMP cranes. Furthermore, an identification model for AC asynchronous motors is established. On this basis, a trajectory tracking method including an adaptive dead zone compensation (ADC) and a modified Smith predictor (MSP) is proposed. Finally, experiments are carried out on a self-built 2-ton crane experimental platform, which is actuated by AC asynchronous motors. Hardware experiments demonstrate that triple pendulum cranes can accurately track the planned trajectory, and the DMP residual swing is effectively suppressed.

Addressing Challenges in Dynamic Modeling of Stewart Platform using Reinforcement Learning-Based Control Approach

In this paper, we focus on enhancing the performance of the controller utilized in the Stewart platform by investigating the dynamics of the platform. Dynamic modeling is crucial for control and simulation, yet challenging for parallel robots like the Stewart platform due to closed-loop kinematics. We explore classical methods to solve its inverse dynamical model, but conventional approaches face difficulties, often resulting in simplified and inaccurate models. To overcome this limitation, we propose a novel approach by replacing the classical feedforward inverse dynamic block with a reinforcement learning (RL) agent, which, to our knowledge, has not been tried yet in the context of the Stewart platform control. Our proposed methodology utilizes a hybrid control topology that combines RL with existing classical control topologies and inverse kinematic modeling. We leverage three deep reinforcement learning (DRL) algorithms and two model-based RL algorithms to achieve improved control performance, highlighting the versatility of the proposed approach. By incorporating the learned feedforward control topology into the existing PID controller, we demonstrate enhancements in the overall control performance of the Stewart platform. Notably, our approach eliminates the need for explicit derivation and solving of the inverse dynamic model, overcoming the drawbacks associated with inaccurate and simplified models. Through several simulations and experiments, we validate the effectiveness of our reinforcement learning-based control approach for the dynamic modeling of the Stewart platform. The results highlight the potential of RL techniques in overcoming the challenges associated with dynamic modeling in parallel robot systems, promising improved control performance. This enhances accuracy and reduces the development time of control algorithms in real-world applications. Nonetheless, it requires a simulation step before practical implementations.

Networked and Deep Reinforcement Learning-Based Control for Autonomous Marine Vehicles: A Survey

Networked and Deep Reinforcement Learning-Based Control for Autonomous Marine Vehicles: A Survey

Training reinforcement learning-based controller using performance simulation of the laser remelting process

Surface topography modification is a common post-processing step used to enhance surface quality or to add micro/nano surface structures for various functionalities. Laser remelting (LRM) is a new technology aimed to resolve some of the shortcomings of traditional post-processing technologies. In LRM, a high-energy laser beam melts the top surface and offers control of the molten material flow through the control of laser power, scanning speed, melt pool size, and scan trajectory. By controlling the molten material flow, surface polishing by laser remelting (SP-LRM) can be used to produce finishes characterized by areal arithmetical mean height - Sa < 0.1 µm as well as surface microstructures. LRM is faster than traditional methods, creates no harmful byproducts, and allows for polishing high aspect ratio freeform surfaces without harming surrounding areas. The primary limiting factor for wide-rage industrial use of the LRM is represented by thermodynamic process instabilities that are associated with rapid melting and solidification. For instance, deep trenches or grinding marks in the initial surface can cause small deviations of local surface absorptivity to lead to laser melt pool instabilities. The high-speed nature of the LRM process combined with the complex thermodynamic phenomena makes process instabilities difficult to model, detect, predict, and control. This study aimed to use reinforcement learning (RL) in order to build a ‘self-learning’ controller capable of adapting to any material and control instabilities on-line and in real-time. Along these lines, the study will detail a preliminary application of two RL implementations involving SP-LRM performance simulation to demonstrate the implementation of a double dueling deep Q-network (DDDQN) controller for melt pool temperature distribution control. The results obtained showed that the 3-action implementation achieved 48% less variance and 35% increase in the mean value of the total reward collected compared with the 31-action implementation.

Reinforcement learning-based control for the thermal management of the battery and occupant compartments of electric vehicles

The coupling between the direct-cooled battery thermal management system and the occupant compartment is difficult to control. Therefore, a control strategy based on reinforcement learning is designed to achieve good control results.

Adversarial deep reinforcement learning based robust depth tracking control for underactuated autonomous underwater vehicle

In this paper, an adversarial deep reinforcement learning-based control method is proposed to address the issue of robust depth tracking of an underactuated autonomous underwater vehicle in the presence of intrinsic coupled dynamics and external disturbances. First, long-short-term-memory neural network is presented to memorize and predict the changes in the state of vehicle, and a cascaded multilayer perception projects the output into action space of vehicle. Subsequently, adversarial deep reinforcement learning scheme is applied to the training of control agent by introducing an adversary which counteracts the control behavior, whereby the agent is enabled to learn the control strategy in different distributions of state transition. For evaluation of the performance, a control agent is pre-trained in simulation environment based on the reliable digital model of a real vehicle, and the simulation environment is paced at one iteration per second to align with real-time operations to ensure the portability of training result. Furthermore, the training cost is also extremely reduced. Finally, experiments are conducted with time-varying disturbances to further prove the feasibility of the proposed learning-based control scheme on a prototype of underwater vehicle in towing tank. Moreover, comparative experiment results show the better robustness performance of the learning-based control agent than that of classic line-of-sight based proportional–integral–derivative and adaptive line-of-sight based proportional–integral–derivative controllers in different scenarios.

Deep reinforcement learning-based control of chemo-drug dose in cancer treatment

Background and objectiveAdvancement in the treatment of cancer, as a leading cause of death worldwide, has promoted several research activities in various related fields. The development of effective treatment regimens with optimal drug dose administration using a mathematical modeling framework has received extensive research attention during the last decades. However, most of the control techniques presented for cancer chemotherapy are mainly model-based approaches. The available model-free techniques based on Reinforcement Learning (RL), commonly discretize the problem states and variables, which other than demanding expert supervision, cannot model the real-world conditions accurately. The more recent Deep Reinforcement Learning (DRL) methods, which enable modeling the problem in its original continuous space, are rarely applied in cancer chemotherapy. MethodsIn this paper, we propose an effective and robust DRL-based, model-free method for the closed-loop control of cancer chemotherapy drug dosing. A nonlinear pharmacological cancer model is used for simulating the patient and capturing the cancer dynamics. In contrast to previous work, the state variables and control action are modeled in their original infinite spaces to avoid expert-guided discretization and provide a more realistic solution. The DRL network is trained to automatically adjust the drug dose based on the monitored states of the patient. The proposed method provides an adaptive control technique to respond to the special conditions and diagnosis measurements of different categories of patients. Results and conclusionsThe performance of the proposed DRL-based controller is evaluated by numerical analysis of different diverse simulated patients. Comparison to the state-of-the-art RL-based method, which uses discretized state and action spaces, shows the superiority of the approach in the process and duration of cancer chemotherapy treatment. In the majority of the studied cases, the proposed model decreases the medication period and the total amount of administrated drug, while increasing the rate of reduction in tumor cells.

50% reduction in energy consumption in an actual cold storage facility using a deep reinforcement learning-based control algorithm

This study presents a unique application of a temperature control algorithm, specifically modified deep deterministic policy gradient (DDPG), in an actual 2.8 m2 cold storage facility, contrasting the majority of research that leverages theoretical validations using simulation tools. The primary goal was to minimize energy consumption while maintaining the desired temperature range. To achieve this, thermocouples and a watt-hour meter were installed to collect real-time data on temperature and power consumption, subsequently transmitted to a deep-learning computing and control system for processing. Utilizing the gathered data, the algorithm was trained to simultaneously maintain the temperature and minimize power consumption. The temperature setting served as a control variable, and a deep deterministic policy gradient algorithm was used. A hyperparameter with a dominant influence on learning outcomes was optimized. Furthermore, the algorithm was exposed to various complex scenarios that occur during actual cold storage operations, such as door opening, reinforcing its practical viability. The study findings revealed that our real-world application of the DDPG algorithm significantly reduced energy consumption by 47.64% compared to conventional proportional-integral-derivative control algorithms, whilst maintaining the target temperature range.

Battery and Hydrogen Energy Storage Control in a Smart Energy Network with Flexible Energy Demand Using Deep Reinforcement Learning

Smart energy networks provide an effective means to accommodate high penetrations of variable renewable energy sources like solar and wind, which are key for the deep decarbonisation of energy production. However, given the variability of the renewables as well as the energy demand, it is imperative to develop effective control and energy storage schemes to manage the variable energy generation and achieve desired system economics and environmental goals. In this paper, we introduce a hybrid energy storage system composed of battery and hydrogen energy storage to handle the uncertainties related to electricity prices, renewable energy production, and consumption. We aim to improve renewable energy utilisation and minimise energy costs and carbon emissions while ensuring energy reliability and stability within the network. To achieve this, we propose a multi-agent deep deterministic policy gradient approach, which is a deep reinforcement learning-based control strategy to optimise the scheduling of the hybrid energy storage system and energy demand in real time. The proposed approach is model-free and does not require explicit knowledge and rigorous mathematical models of the smart energy network environment. Simulation results based on real-world data show that (i) integration and optimised operation of the hybrid energy storage system and energy demand reduce carbon emissions by 78.69%, improve cost savings by 23.5%, and improve renewable energy utilisation by over 13.2% compared to other baseline models; and (ii) the proposed algorithm outperforms the state-of-the-art self-learning algorithms like the deep-Q network.

Tracking Photovoltaic Power Output Schedule of the Energy Storage System Based on Reinforcement Learning

The inherent randomness, fluctuation, and intermittence of photovoltaic power generation make it difficult to track the scheduling plan. To improve the ability to track the photovoltaic plan to a greater extent, a real-time charge and discharge power control method based on deep reinforcement learning is proposed. Firstly, the photovoltaic and energy storage hybrid system and the mathematical model of the hybrid system are briefly introduced, and the tracking control problem is defined. Then, power generation plans on different days are clustered into four scenarios by the K-means clustering algorithm. The mean, standard deviation, and kurtosis of the power generation plant are used as the features. Based on the clustered results, the state, action, and reward required for reinforcement learning are set. In the constraint conditions of various variables, to increase the accuracy of the hybrid system for tracking the new generation schedule, the proximal policy optimization (PPO) algorithm is used to optimize the charging/discharging power of the energy storage system (ESS). Finally, the proposed control method is applied to a photovoltaic power station. The results of several valid experiments indicate that the average errors of tracking using the Proportion Integral Differential (PID), model predictive control (MPC) method, and the PPO algorithm in the same condition are 0.374 MW, 0.609 MW, and 0.104 MW, respectively, and the computing time is 1.134 s, 2.760 s, and 0.053 s, respectively. The consequence of these indicates that the proposed deep reinforcement learning-based control strategy is more competitive than the traditional methods in terms of generalization and computation time.

Deep reinforcement learning-based controller for dynamic positioning of an unmanned surface vehicle

Dynamic positioning (DP) system is of great significance for the unmanned surface vehicle (USV) to achieve fully autonomous navigation. Traditional control schemes have problems such as model accuracy, parameter tuning, and complex design. In addition, although the deep reinforcement learning (DRL) is widely used in the field of vessel motion control, the learning efficiency is not high, and insufficient robustness in the face of changing environmental. In order to improve the anti-disturbance ability, robustness and convergence speed of the controller during training, a deep reinforcement learning control method based on priority experience replay (PER) is proposed for dynamic positioning of the USV. The mathematical models are established based on the kinematic and dynamic of the USV. Markov decision process (MDP) models are constructed according to the DP tasks. The simulation results show that compared with other DRL algorithms, the proposed method has higher reward value, faster convergence speed, higher control precision and smoother control output.

Meta-Reinforcement-Learning-Based Current Control of Permanent Magnet Synchronous Motor Drives for a Wide Range of Power Classes

Data-driven reinforcement learning-based controller schemes have much potential to aid the design of model-free control algorithms that can be trained without the necessity of plant-specific parameter knowledge. Unfortunately, the corresponding training phase is a time- and possibly money-consuming process which needs to be repeated whenever application to a new plant system is requested. To reduce the total training time for a large set of heterogeneous plant systems, this article proposes a meta-reinforcement learning-based approach that is to be utilized for control of hundreds of different permanent magnet synchronous motor drives ranging from a few watts to hundreds of kilowatts. So-called context variables carry the meta information about the set of considered drive systems. Their estimation using a corresponding artificial neural network as context approximator is a core aspect of this article. The context information allows the reinforcement learning-based control algorithm to automatically adapt itself to individual motor drives without requiring individual plant training. Since the found context variables can also be interpreted as an implicit system identification result they allow to determine irregular plant behavior (e.g., faulty drives) as an added bonus of the proposed meta-reinforcement learning scheme. Empirical results during this proof of concept successfully validate the potential of the proposed approach to drastically reduce the total training time and encourage further research.

Reinforcement learning-based control with application to the once-through steam generator system

A reinforcement learning framework is proposed for the control problem of outlet steam pressure of the once-through steam generator(OTSG) in this paper. The double-layer controller using Proximal Policy Optimization(PPO) algorithm is applied in the control structure of the OTSG. The PPO algorithm can train the neural networks continuously according to the process of interaction with the environment and then the trained controller can realize better control for the OTSG. Meanwhile, reinforcement learning has the characteristic of difficult application in real-world objects, this paper proposes an innovative pretraining method to solve this problem. The difficulty in the application of reinforcement learning lies in training. The optimal strategy of each step is summed up through trial and error, and the training cost is very high. In this paper, the LSTM model is adopted as the training environment for pretraining, which saves training time and improves efficiency. The experimental results show that this method can realize the self-adjustment of control parameters under various working conditions, and the control effect has the advantages of small overshoot, fast stabilization speed, and strong adaptive ability.

Deep reinforcement learning-based eco-driving control for connected electric vehicles at signalized intersections considering traffic uncertainties

Eco-driving control poses great energy-saving potential at multiple signalized intersection scenarios. However, traffic uncertainties can often lead to errors in ecological velocity planning and result in increased energy consumption. This study proposes an eco-driving approach with a hierarchical framework to be leveraged at signalized intersections that considers the impact of traffic uncertainty. The proposed approach leverages a queue-based traffic model in the upper level to estimate the impact of traffic uncertainty and generate dynamic modified traffic light information. In the lower level, a deep reinforcement learning-based controller is constructed to optimize velocity subject to the constraints from the traffic lights and traffic uncertainty, thereby reducing energy consumption while ensuring driving safety. The effectiveness of the proposed control strategy is demonstrated through numerous simulation case studies. The simulation results show that the proposed method significantly improves energy economy and prevents unnecessary idling in uncertain traffic scenarios, as compared to other approaches that ignore traffic uncertainty. Furthermore, the proposed method is adaptable to different traffic scenarios and showcases energy efficiency.

DeepValve: Development and experimental testing of a Reinforcement Learning control framework for occupant-centric heating in offices

Space heating controls in offices usually follow static schedules detached from actual occupancy, which results in energy waste by unnecessarily heating vacant offices. The uniqueness of stochastic occupancy profile and thermal response time of each office are two main challenges in hard-programming a transferrable control logic that can adapt space heating schedule to the occupancy profile. This study proposes a Reinforcement Learning-based control framework (called DeepValve) that learns by itself how to adapt the space heating schedule to the occupancy profile in each office to save energy while maintaining comfort. All the aspects of the proposed framework (design, training, hardware setup, etc.) are centered on ensuring that it can be implemented on many offices in practice. The methodology includes three main steps: training on a wide variety of simulated offices with real-world occupancy data, month-long tests on three simulated offices, and day-long experimental tests in an environmental chamber. Results indicate that the agent can quickly adapt to new offices and save energy (40% reduction in total temperature increment) while maintaining occupant comfort. The results highlight the importance of occupant-centric control in offices.